Concurrent tasks #101
Merged
Concurrent tasks #101
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
robinheghan
requested changes
May 12, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
9 times, most recently
from
5a0a78d to
dbf4acb
Compare
robinheghan
reviewed
May 13, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
3 times, most recently
from
880f5f9 to
2d3dccd
Compare
robinheghan
reviewed
May 13, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
3 times, most recently
from
44d3ae6 to
697858c
Compare
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
697858c to
4a40824
Compare
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
4a40824 to
3c0ac32
Compare
robinheghan
requested changes
May 16, 2025
There was a problem hiding this comment.
I think we can do a little better on the documentation, but the implementation and the tests look good to me.
Might be worth waiting on @blaix 's input before making changes.
|
Would you mind adding your name to |
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
3c0ac32 to
47e99c8
Compare
robinheghan
reviewed
May 17, 2025
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
47e99c8 to
f9905ea
Compare
The scheduler queue starts to become very slow when a large number of tasks are in flight (Array.shift is O(n) as it needs to reindex the array every time an element is removed). This modifies `_Scheduler_enqueue` to loop through the active procs and resetting the queue when done. Larger arrays (1M+) are now handled more efficiently.
This runs an array of tasks concurrently. If any of them fail the rest of the running tasks are cleaned up.
Because `map2` is now concurrent `Task.sequence` needs to be implemented using `andThen`.
map4+ become less useful as these are just chains of `andMap`. `andMap` runs concurrently as it's defined using `map2`.
andrewMacmurray
force-pushed
the
concurrent-tasks
branch
from
f9905ea to
829df4d
Compare
|
Thank you for your hard work on this, @andrewMacmurray <3 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds concurrent tasks to gren. The existing
Taskapi has been modified to:Task.concurrent, which runs an array of tasks concurrently.map2run concurrently.map4,map5in favour ofandMap.The existing
Schedulerkernel implementation is largely unchanged but with the addition of a_Scheduler_concurrenthelper that spawns multiple processes, handles collecting results and signalling errors.